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ABSTRACT 


This  paper  considers  a  class  of  variable  metric  methods  for  uncon¬ 
strained  minimization.  The  update  formulas  are  such  that  the  quasi-Newton 
equation  is  not  necessarily  satisfied.  Under  appropriate  assumptions  on  the 
function  to  be  minimized,  each  algorithm  in  this  class  converges  globally 
and  superl inearly.  _  >  \ u 
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SIGNIFICANCE  AND  EXPLANATION 


■  ,-j 


Many  practical  problems  in  operations  research  may  be  reduced  to  mini¬ 
mizing  a  function  with  or  without  constraints.  By  means  of  penalty  functions 
and  similar  techniques  a  constrained  minimization  problem  can  be  converted 
into  a  sequence  of  unconstrained  minimization  problems.  In  this  paper  we  dis¬ 
cuss  a  class  of  algorithms  for  unconstrained  minimi zati j)n^problems  which  con¬ 
verge  rapidly  to  the  solution  from  a  starting  point  which  is  not  necessarily 
a  good  approximation  to  the  solution  of  the  given  problem. 


The  responsibi 1 i ty  for  the  wording  and  views  expressed  in  this  descriptive 
summary  lies  with  MRC,  and  not  with  the  author  of  this  report. 


^ - 


(■  > 


p  _ 


GLOBAL  AND  SUPERL INEAR  CONVERGENCE  OF  A  CLASS  OF 


SCALED  VARIABLE  METRIC  METHODS 


Klaus  Ritter 


1.  Introduction 

Variable  metric  methods  are  successfully  used  for  iteratively  calcu¬ 
lating  a  sequence  {  }  which  converges  rapidly  to  a  global  minimizer  of 

a  smooth  convex  function  F(x)  .  At  each  point  Xj  the  computation  of  the 
search  direction  is  based  on  a  quadratic  model  Q(x)  which  interpolates  the 
function  value  and  the  gradient  of  F(x)  at  x^  as  well  as  the  gradient  of 
F(x)  at  Xj_j  .  In  general  Q(Xj_^  *  F ( x ^ ^ )  .  The  matrix  defining  the 
quadratic  form  in  this  model  is  updated  in  such  a  way  that  it  satisfies  the 
quasi-Newton  equation. 

In  this  paper  a  class  of  modified  update  formulas  is  considered  which 
result  in  a  quadratic  model  Q(x)  which  in  addition  to  interpolating  the 
function  value  and  the  gradient  of  F(x)  at  x^_^  minimizes  the  criterion 

til  v  II  2  +  ( 1  -  t)(Q(Xj_1)  -  Ftx^))2  . 

Here  v  is  the  error  in  the  quasi-Newton  equation  and  t,  0  <  t  <  1,  is 
a  weight  factor  which  reflects  the  importance  assigned  to  satisfying  the 
quasi-Newton  equation  as  opposed  to  interpolating  F(x^_j)  .  Under  the  usual 
assumptions  it  is  shown  that  the  resulting  variable  metric  methods  converge 
globally  and  superl inearly. 
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The  so-called  restricted  Broyden  methods  [2]  are  contained  in  this  new 
class  of  variable  metric  methods.  They  are  obtained  if  the  weight  factor  t 
is  chosen  equal  to  one. 

In  [1]  Biggs  describes  a  modification  of  the  Broyden-Fletcher-Goldfarb- 
Shanno  method  which  results  in  an  update  formula  for  which  the  quasi-Newton 
equation  is  also  not  satisfied.  However,  this  method  is  not  contained  in  our 
class.  Recently,  Schnabel  [7]  has  shown  that  Biggs'  method  converges  super- 
1 inearly. 


2.  Basic  properties  of  scaled  variable  metric  methods 

Let  x  e  En  and  let  F(x)  be  a  real  valued  function.  If  F(x)  is 
twice  differentiable  at  some  point  x.  we  denote  the  gradient  and  the  Hessian 

J 

matrix  of  F(x)  at  x-  by  7F(x.)  =  g.  and  G-  =  G(x.)  ,  respectively.  A 

J  J  J  J  J 

prime  is  used  to  denote  the  transpose  of  a  vector  or  a  matrix.  For  every 
x  £  En  ,  II  x  ||  denotes  the  Euclidean  norm  of  x  . 

Throughout  this  paper  we  make  the  following 
Assumpt ion  1 

F(x)  is  twice  continuously  differentiable  and  there  are  numbers 
0  <  u  <  n  such  that 

u  II  x  ||^  £  x'G(y)x  ^  n  II  x  11^  for  all  x,  y  €  En  . 

It  is  well-known  that  Assumption  1  implies  that  F(x)  is  uniformly 
convex  and  that  there  is  a  unique  z  €  En  such  that 

-  2  - 


(2.3) 


where  G  =  G(z)  and 


(2.4) 


E.  =  /  G(x.  -  t  OjS . )dt  -  G 

j  o  J  j  J 


Dividing  (2.3)  by  II  g^  II  and  using  the  triangle  inequality  we  obtain 
the  two  inequalities 


"jSJ 


(2.5) 

".Vi"  < 

9J _ G 

F 

II  g,  II 

ll  g,  II  II  9;  II 

J 

II  9j  II 


and 


(2.6) 


!l  -  G  -Z£l 


ej 


qjsj 

II  g -II  i! 

J 


Since 


E.  II  <  II  /  G  (x.  -  to -  s  .  )dt  -  G(x.) 

J  g  J  J  J  J 


+  II  G(Xj)  -  G 


implies  that 


(2.7) 


E  .  II  -*■  0  as  j  - 


and  it  follows  from  Assumption  1  that 


°jsj 11  ■  °<"  V> 


we  deduce  from  (2.5),  (2.6)  and  (2.7)  that  the  sequence  f g ^ }  converges  super- 
linearly  to  zero  if  and  only  if 


G  - 


°jSj 


-  0  as  j 


This  shows  that  in  order  to  achive  superl inear  convergence  we  have  to 


determine  the  search  direction  Sj  and  the  step  size  Oj  in  such  a  way  that 


(2.8) 

i|  g.  Gs .  Ij 

— 1-  .  .  - -J- —  •  0  as  j 

i  II  gj  II  II  Gs  j  II  il 

and 

(2.9) 

1  II  Gs,  II  ; 

a-  — ■  -  -  1  1  -»  0  as  j 

i  J  II  g,  II  i 

J 

This  procedure  can  be  motivated  as  follows.  If  Hj  approximates  G 
in  the  sense  that 

(2.13)  II  H.  -  G_1  II  -*■  0  as  j  -  »  , 

0 

then  it  is  clear  that  Sj  as  defined  by  (2.10)  satisfies  (2.8).  Furthermore, 
it  follows  from  (2.3)  that 

(2.14)  G_1dj  =  Pj  +  G_1E.p.  . 

Thus  if  Hj+j  satisfies  the  quasi-Newton  equation  (2.11)  we  have 


-  5  - 


'Vi  -  G‘ )dj 


0  as  j 


It  should,  however,  be  observed  that  the  condition  (2.13)  is  not  neces¬ 


sary.  According  to  (2.8)  we  only  need  the  property 

>0  as  j  - 


"  jr’9! 

fi  IIG-T9j 


H  .g  ■ 
JSJ 


II  H j g j  II 


combined  with  a  step  size  satisfying  (2.9). 


Assuming  that  is  symmetric  and  positive  definite  we  denote  the 

inverse  of  by  and  consider  the  quadratic  function 

(2.15)  Q(x)  -  F(xj  +  1)  +  gt  +  1(x-xj+1)  +  ^  (x-xj+1)’Bj+1(x-xj+1)  . 

Obviously  we  have 


(2.16)  Q(*j+1)  =  F(xj+1)  and  7Q(xj+i)  = 


j+1  ' 


Furthermore,  if  Hj  +  j  satisfies  the  quasi-Newton  equation  (2.11)  then 


(2.17) 

and  it  follows  that 


(2.18) 


VQ(Xj) 


,-l 


B  .  ,  p  .  =  d  . 

J  +  l^J  J 


9-.i+B.,(x.-x.,)  =  g. 

aJ+l  J+1  j  j  +  1' 


With  s  .+j  =  Bj  +  1gj+1  it  is  easy  to  verify  that  x^+1  -  Sj  +  1  is  the 
unique  global  minimizer  of  the  quadratic  function  (2.15).  Since  B  =  Hj+j 
we  can  interprete  the  negative  search  direction  -  s^.  +  ^  =  -  H^g^  ,  asso¬ 
ciated  by  a  variable  metric  method  with  a  point  x^+j  ,  as  the  direction  which 
leads  from  Xj  +  j  to  the  global  minimizer  of  the  quadratic  function  (2.15) 
which  has  the  properties  (2.16)  and  (2.18). 


/  .  v  .  \ 


J 


Since  in  general  0 ( x  - )  *  F(x.)  we  could  try  to  determine  the  quad- 

J  J 

ratic  function  Q(x)  such  that 

(2-19)  Q(Xj)  =  F( Xj ) 

by  giving  up  one  of  the  properties  (2.16)  or  (2.18).  Since  at  x.+1  the 

function  value  F(Xj+^)  and  the  gradient  represent  the  most  recent 

information  about  F(x)  that  is  available  to  us,  it  appears  reasonable  to 
insist  on  (2.16).  However,  then  the  quasi-Newton-equation  (2.17)  necessarily 
leads  to  the  equality  (2.18).  Therefore,  if  we  want  to  use  a  quadratic  func¬ 
tion  of  type  (2.15)  that  has  the  properties  (2.16)  and  (2.19)  we  have  to  re¬ 
lax  the  quasi-Newton  equation  (2.17). 

Multiplying  the  equality  (2.14)  by  G  we  obtain 

Gpj  -  dj  +  Ejpj  ■ 

Thus  the  quasi-Newton  equation  (2.17)  guarantees  that 
II  Bj  +  1Pj  -  Gpj  II  -  0  as  j  -  00  . 

However  since  Gp^  differs  in  general  from  d-  there  is  really  no  reason 

to  insist  on  (2.17).  On  the  contrary  it  might  be  better  to  relax  (2.17)  in 

favor  of  a  better  approximation  of  F(x-)  by  Q(x-)  . 

J  J 

In  the  following  we  will,  therefore,  consider  a  quadratic  function  (2.15) 
where  Bj+1  is  symmetric  positive  definite  and  satisfies  the  relaxed  quasi- 
Newton  equation 


We  assume  that  Vj  is  an  optimal  solution  to  the  problem 

-  7  - 


(2.20) 


(2.21) 


min  {  t  II  v  II  2  +  ( 1  -  t)  (Q(Xj )  -  F(x^  j  )2 
v 


Here  the  parameter  t  is  restricted  to  the  interval  0  t  1  .  It  can  be 
interpreted  as  a  weight  factor  which  reflects  the  importance  we  assign  to  sa¬ 
tisfying  the  quasi-Newton  equation  as  opposed  to  interpolation  F(Xj)  . 


Using  (2.20)  we  obtain  from  (2.15) 


(2.22)  Q(Xj)  =  F( Xj+1 )  ♦  g-.+1(Xj  -  xj+1)  *  ?  (xr  xJ+1)-(9j  -  gjtl> 

+  l,|xj-xj+i11  <Vxj+i),vj 

’  F(xj*l>  *1  <VW(VXJ+1)  +lllXj-xj+1ll2p}vj 


Setting 


(2.23)  vj  ■  F(xjtl)  -  F(xj)  (gj*9j+1)'(xrx.+1) 
we  can  therefore  write  (2.21)  in  the  equivalent  form 

(2.24)  min  {  t||v||2  +  ( 1  -  t)(vj  +  j  II  -  xj>+1 1| 2  pjv)2  }  . 


Clearly  v.  is  an  optimal  soltuion  to  (2.24)  if  and  only  if 

J 

(2.25)  tvJ+  (l-t)(vJ.*l|lxj-xJ+1ir2pi,j)pj  =  0. 

If  t  >  0  this  implies  v,  =  X.p,  for  some  A.  .  In  order  to  obtain 

J  J  J  J 

substitute  into  (2.25)  and  solve  for  A.  .  This  gives 

J 


Therefore, 


= 


2 ( t  -  l)vj 


2t  +  (1  -  t)  II Xj  -  Xj+1l 


(2.26) 


2(t-  l)v, 


J  2t  +  ( 1  -  t)  II Xj  -  Xj+1 


ii  2"  PJ 


-  8  - 


, 


is  an  optimal  solution  for  (2.24).  If  t  0  ,  then  '2.26)  is  the  unique 


optimal  solution.  Furthermore,  if  t  =  1  ,  then  v ^  =  0  and  (2.20)  reduce' 
to  the  quasi-Newton  equation  (2.17).  On  the  other  hand  if  t  =  0  ,  then 


-2,. 

J 

x  •  -  x  .  ,  | 
J  J  +  1 


TT  pi 


and  it  follows  from  (2.22)  and  (2.23)  that  Q(x-)  =  F(x.)  . 

J  J 


(2.27) 


6.  =  6  Jt) 


2t  +  (1  -  t)  II Xj  -  x.  +  l\ 


we  obtain  from  (2.20)  and  (2.26)  the  relaxed  quasi-Newton  equatii 


B  .  .  p  .  =  d.+6-p.  , 


or  in  terms  of  Hj+j  , 


(2.28) 


Vi(dj +  W  =  pj  ■ 


By  Taylor’s  theorem  there  are  vectors 


(2.29)  Zj  ,  2^  6  {  x  1  x  =  xj+1  ♦  Mxj-x^)  ,  0  <  >  <  1 


such  that 


F(Xj)  x  F(xJtl)  +  gj+j(Xj  -  »jtl)  +  7  (xj  -  xj+l>'G0J)(Xj  -  Xj  +  1 ) 


9j(xj  ■  Xj  +  1>  =  9j+l(Xj  ■  xj+l>  *  (Kj  '  xj*l),G|zj)<Xj  -  Xjtl) 


Therefore,  it  follows  from  (2.23)  that 


'j  =  2  iy»i(i),<^J)-«ii)Hy«i(i) 


which  by  2.27)  implies  that  there  is  a  constant  •  such  that 


(2.30) 


■j  (t)  _  (1  -  t)  5  II  G(  z  j )  -  G(  Zj  )  II  ,  0  t  <  1  . 


Setting 


d  .  =  d  •  +  6  .p  . 

J  J  J  J 


2  2 

choosing  arbitrary  parameters  £,  and  B?  with  6^  +  ^  0  we  define 


the  following  class  of  update  formulas 


S1(d'.pi  +  d'.H  d.)  +  Bod.'H.d. 

(2.31)  H  =  H.  +  j.J - ill - p  p; 

<VjVWVj)dj,pj  “ 


p.d'.H.  +  H.d.p.'  H.d.d.'H. 

.  ,  J  J  J _ J  JHJ  .  e  J  J  J  J _ 

1  Bid.'  p,  +  Bpd‘.H,d.  ^  B i d - ’  p -  +  8?d,'H,d, 

J  J  ^  tj  sj  J  ^  J  J  J  J  J 


Assuming  that  all  denominators  in  (2.31)  are  different  from  zero  it  is  easy 
to  verify  that  each  matrix  Hj  +  ^  defined  by  (2.31)  satisfies  the  equation 
(2.28).  Moreover,  if  Hj  is  symmetric  so  is  . 


If  we  choose  t  =  1  ,  i.e.  dj  =  dj  ,  then  (2.31)  is  identical  with 
a  class  of  update  formulas  introduced  by  Broyden  (see  [2]  and  [3]). 


In  order  to  determine  conditions  on  Bj  and  which  guarantee  that 
the  update  formula  (2.31)  maintains  the  positive  definiteness  of  Hj  we  de¬ 
fine  the  two  subspaces 

Sj  ■  spa"  '  5j.  dj  1 

j,  ■  '<  <Vj>'x  s  0 •  (Hjdj),x  =  0 1 

and  observe  that 


>,2.32) 


H  j  +  lx  =  Hjx  f°r  x  €  T j  . 


Since  g.  i  T-  and  d.  {  T.  it  follows  that  H.  ,  is  completely  de- 

J  J  J  J  J  ^ 

fined  if  we  specify  it  on  the  subspace  Sj  .  For  each  update  formula  (2.31) 
we  have 


(2.33) 


HMdj  '  pj  ■ 


If  and  d^  are  linearly  dependent,  then  l-L  +  ^  is  completely  defined  by 

(2.32)  and  (2.33).  If  and  dj  are  linearly  independent  let  Wj  £  Sj  be 


such  that 


wip,  =  0  aiad  H.w.  =  q .  with  II  q.  II  =  1  . 

J  J  J  J  J  J 


d'.H.  ,w.  =  piw.  =  0 

J  J+l  J  FJ  J 


H .  ,w . 

J+l  J 


.  <eiVs2Hjclj>li.iI’,i 

"ltyj  *  Wj 


€  span  {  q^,  } 


This  implies  that  there  is  a  vector  u.  e  spanfq.,  p.  }  such  that 

J  0  0 


"  “j 11  "  1  •  Vj  ■  0  '  "j“j  =• 0 


and,  for  every  update  formula  (2.31), 


(2.34) 


Viwj  ‘  Vj  ■ 


;re  only  the  parameter  depends  on  the  particular  values  of  Bj  and 


Since  we  assume  that  Hj  is  symmetric  and  positive  definite  it  is  not 
difficult  to  verify  (see  [6])  that  Hj  can  be  written  in  the  form 


(2.35) 


+  H. 
J 


PJPj  ,  qJqJ 
jgjpj  WjQj 


where  .  ■  =  II  H.g,  II  and  H- 

J  J  J  J 


^  and  H.  is  a  matrix  of  rank  n-2  with 


H.g.  =  H -w .  =  0  . 

JyJ  J  J 


Now  it  follows  from  (2.32),  (2.33),  (2.34)  and  (2.35)  that 


(2.36) 


djpj  J 


Therefore,  if  Hj  is  positive  definite,  then  is  positive  definite  if 


and  only  if 


(2.37) 


d  .p  .  >  0  and  co,  >  0  . 
J  J  J 


Because  d.p.  =  d'.p.  +  6-(t)  and  dip.  >  y  (see  Lemma  1)  we  can  force 
J  J  J  J  J  J  J 

d’.p.  to  be  positive  if  we  choose  t  sufficiently  close  to  one  (see  (2.30)). 

J  J 

In  order  to  study  the  dependence  of  oo,  on  and  B^  we  first  choose 
=  1  and  B2  =  0  .  If  t  =  1  the  resulting  update  formula  (2.31)  corre¬ 
sponds  to  the  Broyden-Fletcher-Gol dfarb-Shanno-method  [3],  [4],  [5],  [8].  Then 

d’q. 

Vi“j  *  qj '  aTT  pj  ' 

r  j 


Thus,  setting  a ■  =  d'q,  /  d',p.  we  have 

«J  J  J  \J 


(2.38) 


qj  ~  ajPj 

qj  ‘  Vj 


Wl 


Since  by  (2.35)  , 


» 


(2.39) 


d  '•  p  .  d q  . 

H  d  .  =  p  -  -  -  +  q  -  - 

J  J  J  .  .  g p  .  J  w q  . 


we  obtain  for  a  general  choice  of  the  parameters  8  ^  and  .  ^ 


H.  ,w. 
J+l  J 


q.i(Rldjpj  'B2djHjdj>  -  Vj^j  -  -2Hjdjdjqj 
eldjpj  *  62djHjdj 


1 d j P j  +  62(djpj>  /  f  j9jpj 

SldjPj  +  62djHjdj 


(dj-°jpj)  ' 


Thus  in  general 


(2.40) 


where 


(2.41) 


q  11  dj  •  'jpj 11  - 


eldjp1  +  8;(djPj)  /  Pj9jPj 

eldjPj  *  B2djVj 


This  shows  that  dj p j  >  0  and  >  0  ,  +  $2  *  0  is  a  sufficient 

condition  for  Wj  >  0  ,  which  by  (2.37)  implies  that  is  positive  de¬ 

finite.  If  3^2  <  0  >  then  Yj  could  be  zero  or  negative.  In  this  case  an 
adjustment  of  the  parameters  8]  and  82  is  required  in  order  to  obtain  a 


positive  definite  matrix  Hj+^  . 


For  later  reference  we  prove  the  following  lemma 


Lemma  1 


i )  For  every  x  •  e  E  , 

V# 


g-  II  <  n  II  x,  -  z  II  and  y  ||  x.  -  z  II  <  II  g. 

J  J  J  J 


5^ v 


ii)  Let  d-  and  p  be  defined  by  (2.12).  Then 
J  J 


c'.p 
J  .1 


ird  II  dj  II  < 


Proof: 


i)  By  Taylor's  theorem  there  are  vectors 

z  ■ ,  z  •  €  {  x  x  =  x.  -X(x,-z),  o  i  }  i  1 
J  J  J  J 

such  that 

u  II -  z II 2  <  (Xj  -  z)1  G(Zj )(Xj  -  z)  =  gJ(Xj-z)  <  llgjll  «•  x ^  -  z I 
and 

II g j II 2  =  9jG(Zj)(xj "  7~)  1  n  II  gj  II  H  *j '  z  11  . 

ii)  Using  Taylor's  theorem  again  we  obtain 

d’-p -  =  —  =  pjG(zj)pj  -  u 

J  J  II  a. s.  II  J  J  J 

J  J 

and 

lld.n2  =  lgj  ~  q3+}}Ji  d',G(z  •  )p  •  <  II  G(z  • )  II  lid  II  <  n  lid  .  II 
J  II  a-s-  II  J  J  J 

J  J 


3.  Convergence 


Based  on  the  discussion  of  the  previous  section  we  describe  now  an 
algorithm  which  for  any  starting  point  xQ  and  any  symmetric  positive  de¬ 
finite  matrix  Hq  generates  a  sequence  {  x-  }  which  either  terminates  with 
z  or  converges  superl inear  to  z  . 

At  the  beginning  of  a  general  cycle  of  the  algorithm,  x.  ,  g.  *  0 

J  J 

and  a  symmetric  positive  definite  matrix  are  available. 
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Step  1:  Computation  of  the  search  direction) 
Compute 

s  •  =  H  .  g  . 

J  JyJ 

and  go  to  Step  2. 


Step  2:  'Computation  of  the  step  size  j) 

Determine  ^  such  that 

F( x j  ~  j s j )  =  min  <  F(xj _  a  sj )  1  0  o  )  • 

0 

Set 

x  .  ,  =  x  .  -  a  .  s  . 

J+1  J  J  J 

and  compute  g^+j  .  If  g^  +  ^  =  0  stop,  otherwise  go  to  Step  3. 


Step  3:  (Computation  of  Hj+^) 

Select  0  <  t  <_  1  ,  Sj  and  S£  with  8^  +  >  0  and  compute 

vj  =  F(xj+1>  -  p(xj>  (9j  +  9j+l),(xj“xj+i) 


6  -i  ( t ) 


2(t-  l)v. 


2t  *  (1-t)  NXj-x^j  II 


ij  •  ,  ,  ,t)  -i 

"  Cisj  11  J  " 


Determine  Hj+1  by  formula  (2.31).  Replace  j  with  j+1  and  go  to 
Step  1. 


Remark 

In  Step  2  we  assume  that  ck  is  the  optimal  step  size.  This  assump¬ 
tion  simplifies  the  proof  that  the  sequence  (Xj)  converges  superl inearly  to 
x j ■  .  Using  an  approach  similar  to  the  one  in  [6],  it  can  be  shown  that  this 
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result  remains  true  if  j 


is  an  appropriate  approximation  of  tne  .’0t  a 


step  size.  In  Step  3  we  assume  that,  if  necessary,  the  parameters  t  , 
and  2  are  adjusted  in  such  a  way  that  d'.p.  =  d'.o.  +  .'t)  0  and 

defined  by  (2.41)  is  positive.  As  we  have  seen  in  the  previous  section  this 
implies  that  is  positive  definite. 

In  the  following  lemma  we  establish  some  properties  of  the  sequences 
f  g  j }  and  ( x j  >  generated  by  the  algorithm. 

Lemma_2 

ii )  gtp j  -*  0  as  j  <*> 

Hi )  II  Xj+1  -  Xj  II  -»•  0  as  j  -*•  °°  . 

By  Taylor’s  theorem  we  have,  for  every  a  _>  0  ,  the  inequality 
F(xj  '  0Sj>  1  F(xj)  '  °9jsj  +  \  o2  II  Sj  II2  n  . 

a  ? 

For  o.  =  gls.  /  II  s  -  II  n  this  implies 

J  J  J  J 

F(Wj>  -  F(xj'5jsj>  -  F(xj>  -  ^ 

Because  F(x)  is  bounded  from  below  and  F( x j  +  )  <  F(x^)  for 

every  j  ,  it  follows  from  part  i)  that  g'.p.  ->-0  as  j  -  °c  . 

J  J 

By  Taylor's  theorem  there  is  a  vector  z-  such  that 

3 

Zj  £  {  x  '  x  =  Xj  -  A ( x j  +  ^  -  Xj)  ,  0  <  A  <  1  } 

and 

0  ■  9j*lpJ  ■  SjPj  -  • 
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Proof: 

i) 


ii) 


iii) 


T 


Therefore , 


(3.1) 


plGIZjlPj 


0  as  j 


In  order  to  prove  that  the  sequence  { x ^ }  converges  to  z  we  need 

some  properties  of  the  trace  of  the  matrices  H.  and  B.  .  By  definition, 

J  J 

the  trace  of  a  matrix  is  the  sum  of  its  diagonal  elements,  which  in  turn  is 
equal  to  the  sum  of  the  eigenvalues  of  the  matrix.  Using  the  representation 
(2.35)  of  Hj  we  obtain 


(3.2) 


tr<V 


pjgjpj  Wjq3 


+  tr(H.)  . 


Using  (2.35)  once  more  it  is  not  difficult  to  verify  that  Bj  =  Hj1  can  be 


written  in  the  form 


P^-g  w.w. 
a. .a.  a  +  a  a  +  g 


where  Bj  is  a  matrix  of  rank  n-2  with  the  property 


Vo  ■  Vj  ■  0  ■ 


Thus  we  have 


(3.3) 


tr(Bj) 


'j  11  g3 


!1  w  II' 

J _ 


+  tr(Bj)  . 


Mow  let 


«>j  =  tr(Bj)  +  tr(Hj ) 


Then  it  follows  from  (3.2),  (3.3),  (2.36)  and 

<M  ,  II  w  -  II 2 

B .  ,  =  J-i  +  -L - J -  < 


djpj  “j  wjuj 


+  tr(Bj) 
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that 


'■P 


j  +  1  ~  'Pj 


l.u  j9jH? 

p  4  -  --  -  + 


jgjpj 


1  ♦  II  6j  II' 

d'.p  • 
J\] 


+  i+"W-'  J 


Wjqj 


Vj 


Since  2.38)  and  wjPj  =  0  imply 


j  .  : 


WjUj 


w  .q  . 

JL  2 . 

q  ■  -  ( -p  ■ 
J1  J 


we  can  write  the  above  equality  in  the  form 

,2 

i+IIPigill  i  +  II  ai  i 

(3.5)  .Pj+1  =  ^- 


l+||p.gjl2  1+lld^l2  1  II  Wj  II 2  H9j  ”  ajpj  l|2',j"1 


J 


+  (--!' 


.  q p  . 
JyJKJ 


d'.p  . 
r  J 


Yj  Wjq3 


Wjqj 


By  Lemma  2,  II  x^  +  1  -  II  —  0  as  j  ->•  »  .  Therefore  it  follows  from 

(2.29)  and  (2.30)  that,  for  0  <  t  <  1  , 

H  dj  '  dj  II  =  I  6j(t)  I  -  0  as  j  -  «  . 

In  connection  with  Lemma  1  this  implies  that  there  is  a  constant  u  such 


that 


(3.6) 


1  +  M  dj 
d'.p . 

r  j 


f  uQ  for  j  =  0,1,. . .  . 


Next  we  infer  from  dj  =  (g^  -  Qj  +  j)  /  H  °jsj  II  +  SjPj  that 
j+1  "  "  j'j  "  JrJ 
(3.7) 


g^i  "  II  o^s,-  II  <5^p,.  £  3  ■  Defining 


yj  -  Pj 


"jS.i  "  h  d 
diPj  J 


and  observing  that  p'.(g.  ,  -y.)  =  0  we  deduce  from  the  definition  of  u. 

J  J  J  J 

.  _ 


and  Wj  that 


(3.8) 


VilOj.rq) 


"j  *  ‘  “J 


lVi(qj.i'yj)" 
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Using  (3.4)  and  1.3)  we  obtain 


(3.9) 


l)  ^ 

'  wtq. 


1-v.)  — 


gj  +  l'yj 


^ 9 j  +  l_yj  ^  Hj  +  l^9j  +  l  ’  yj  ^ 


(3.10) 


vl  =  xi 

wiqj  :j  ^j+i"yj>'Hj+i(sj+i’yj) 


where 


(3.11) 


tj  =  VWj 


Unfortunately  it  does  not  appear  to  be  possible  to  find  an  a  priori 
upper  bound  for  the  expressions  on  the  right  hand  side  of  (3.9)  and  (3.10). 
However,  since  it  follows  from  (2.30)  and  (3.7)  that  y.  =  0  if  we  choose 
t  =  1  ,  there  is  t,  <  1  such  that  for  every  t  with  t-  <  t  <  1  the  fol- 

J  J 

lowing  condition  is  satisfied. 


Condition  1 

For  every  j  the  parameter  t  is  determined  such  that  0  <  t  <  1  and 

ii  gi+1  -yi  ii2  ii  gi+1  u2 

(i-Y  )  - J.tA ,j_. - - -  <  u 

^9j+i"yj^  Hj+i^9j+i  ’  yj ^  9j+iHj+i9j+i 


V1  "  Vi(gj+ryj) 


|| H  .  .g  .  ,  | 

J+l^J+1 


j  ^9j  +  l  "  yj ^  Hj+l^9j  +  l  “  yj ^  9j+lHj+l9j  +  l 


-  u2 


where  and  are  arbitrary  positive  constants. 


Using  Condition  1  we  obtain  the  following  lemma 
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Lemma  3 


Let  Condition  1  be  satisfied.  Then  there  is  a  constant 
that 

j  II  9^  II 

I  -  <  (j  +  l)  p  for  j  =  0,1,2,... 

i-o  g;Pj 


Proof: 


For  every  j  we  define  the  matrices  B-(y.)  and  H.(T.)  as 

J  J  J  J 


(3.12) 


Wi 

gjpj 


J  ^  n>n  i.i  I  n  3 


9jPJ 


(3.13) 


w 


Hj'(1  T  • 


P3P3 


j-1  PjgjPj 


1 


p  j  pis  q^i 

+  H. 


"j-1  Pj9jPj  Wjqj  ' 


Setting 


*  trlBjUj)  +  tr(HjCTj)) 


we  obtain,  similar  to  (3.5),  the  equality 

1  (  1 


(3.14)  =  <k-  - 


j  +  l  r  j  -.-IT 


PjgjPj  '  ^ 


7  1  +  II  d, 

Yj-1  ,lpj9j"  + 


d'.p . 
J  J 


1  \  11  wi  I' 

+  I  —  -  1  — L 

'  wjqj 


d-Yj)- 


C  j+l9j  +  l  ' 


°j+l9j+lPj+l 


,  id  .  (1-X, - i 


“jqj 


J  'j+lgj+lPj+l 
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such 


fol 1 ows 


Using  (3.6),  (3.9)  and  (3.10)  and  observing  that 


1 


'  j+1 


j+1  "  j+lHj+lgj+l 


II  Hj+1gj+1  II 

we  deduce  from  (3.14)  and  Condition  1  the  inequality 

1 


(3.15)  ;,j+1  Vj 


Wj 


^  *  0-1  "OO"2)  + 


uo  +  u\  +  y2 


-V-  (tt'o  *  Yi-lllpi9i"2)  *  0+l)(«o+  -l  +  "2>  • 
1  =  1  p.j gi p-j  \  j-i  / 


Since  y-  and  t.  are  positive,  B-(y.)  and  H.(t.)  are  positive 

J  J  J  J  J  J 

definite.  Therefore,  every  v-  is  positive  and  it  follows  from  (3.15)  that 

(3.16)  I  — — —  /  — — —  +  Y •  j  I ! P i 9 i  1 1 ^ )  £  ( J  + 1 )  p 4  ’  J  =  0,1,2,... 

i=0  pig-Pi  \  Li-i  7 

for  some  constant  u*  >  0  . 


Because  Yi_1  >  0  and  ii_1  =  yi_1  II _  1 
verify  that 

1 


'-‘i-lP-j-l  II  it  is  easy  to 


(3.17) 


Pi9ipi  V  i_1 

Moreover,  for  every  j  , 


1  Z\  2llgi" 

-  +  Virgin  >  — ~ 


Vi 


gi-rai-ipi-i' 


-i 


II  d.  II  <  n  + 1<5 j( t) |  and  d^  =  dtp ^  +  6j(t)  >  u  +  rj(t)  • 

Since  by  (2.29),  (2.30)  and  Lemma  2,  6.(t)  -*-0  as  j  -*■  °°  ,  we  have 
d  -  p  -  >  p/2  for  j  sufficiently  large.  Therefore,  it  follows  that  there  is 

J  J 

i  >  0  such  that 

(3. 18)  II  q  •  -  a.p,  n’1  >  e  >  o  for  j  =  0,1,2,. .  .  . 

J  J  J 
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'  "*V«’  M 


The  statement  of  the  lemma  follows  now  from  (3.16),  (3.17)  and  (3.1ft). 


With  Lemma  3  at  our  disposal  we  can  now  easily  prove  the  main  result 
of  this  section 

Theorem  1 

Let  Assumption  1  and  Condition  1  be  satisfied.  Then  the  sequence  ■  Xj 
generated  by  the  algorithm  either  terminates  with  z  after  a  finite 
number  of  iterations  or  converges  to  z  ,  where  z  is  the  global  mini- 
mizer  of  F(x)  . 

Proof: 

It  follows  from  Lemma  3  that  there  is  a  positive  constant  Uc  and  an 
infinite  set  J  c  {0,1,2,...}  such  that 

!l  It 

■  -  —  <  pr  for  j  €  J  . 

9JPj 

Since  by  part  ii)  of  Lemma  2  p'.g.  ->-0  as  j  -*■  °°  this  inequality  shows 

J  J 

that 

(3. 19)  II  g,  II  -  0  as  j  -  «  ,  j  e  J  . 

By  the  continuity  of  7F(x)  this  implies  that  VF(x)  vanishes  at  every 
cluster  point  of  the  subsequence  { x j ,  j  e  J)  .  Since  the  set 
{  x  1  F(x)  <  F(x  )}  is  compact  and  VF(x)  *  0  for  x  *  z  we  deduce, 
therefore,  from  (3.19)  that  a  subsequence  of  f x j }  converges  to  z  . 
Observing  that  F ( x j  +  ^ )  <  F(xj)  and  F(z)  <  F(x)  for  x  *  z  ,  we  see 
that  Xj  -»  z  as  j  -  ®  . 
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Super! i near  convergence 


In  order  to  show  that  the  sequence  x-  converges  super! inearly  to 

J 

z  we  need  the  following  additional  assumption. 


Assumption  2 

There  is  a  positive  constant  L  such  that 


G(x)  -  G(y)  II  <  L II x  -  y  II  ,  x,  y  e  En 


Let  G1/2  denote  the  square  root  of  G  =  G(z)  and  set  G~1//2  =  (G^2)"'*'. 
As  a  first  result  we  will  show  that  with  an  appropriate  modification  of  Condi¬ 
tion  1  the  sequence  {  }  , 

J 

Cj  =  tr(G"1/2B(Yj)G"1/2)  +  tr(G1/2Hj (, j )G1/2) 

is  bounded,  where  the  matrices  B(y.)  and  H.(y.)  are  defined  by  (3.12)  and 

J  J  J 

(3.13),  respectively. 

Observing  that  by  (2.28),  (2.40)  and  (3.4) 


ujGui 

J  J 


_ a.p . i,2  u1Guj  , 


it  is  not  difficult  to  show  that  in  analogy  to  (3.14)  we  have 


(4.1) 


1  /p  ;-GP,  O  1  \  piGp,  +d'.G_1d. 


'  wjqj  J  gj+lpj+l 

+  (Yj-1)  II q i  -  a  .p  •  II 2  "A-  -  (1-  l  )  _ 

Wjqj  :j  Pj+lgj+lPj+l 

,  (qj-ajpj)'G(qj-^P.j)  .  0j<*j 


/' 


In  deriving  estimates  for  the  terms  on  the  right  hand  side  of  the 
above  equality  we  will  use  a  result,  derived  in  [6],  which  states  that  for 
every  x,  y  €  En  with  y‘ x  >  0  the  following  relation  holds 


(4.2) 


x'Gx  +  y'G_1y 


=  2  + 


(y  -  Gx) ' G" x (y  -  Gx ) 


Lemma  4 


Proof: 


p'jGp,  +  d,G  1d.  2 

J - -  2  =  0(llx.-zir 

d'.p .  J 

J  J 


It  follows  from  (2.3)  and  (2.4)  that 


(4.3) 


9-j  " 

- - 111  *  Gp  *  EjP, 

II  C  II  J  J  J 


(4.4) 


E.  II  =  11  J  G  (x,  -  to.s  .)dt  -  G 

J  _  J  J  J 


<  II  /  G  ( x  -  -  tc.s.)dt  -  G .  II  +  II  G.  -  G 

fX  J  J  J  J  J 


<  max  {  II  G(x .  -  to.s  •)  -  G-  II  +  II  G-  -  G  II 


0<  t<  1 


J  J  J  J 


<  L  II  Xj  +  1  -  Xj  II  +  L  ||  Xj  -  z 


<  L  If  Xj+1  -  z  II  +  2L  II  x  .  -  z  II 


=  0  (  II  Xj  -  z  II  )  , 


where  the  last  relation  follows  from  (2.5)  and  part  i)  of  Lemma  1. 


Since  by  (2.29)  and  (2.30) 


(4.5) 


I5j(t)(  =  <  6(1-  t)  L  II  *i  +  1-Xj  II  =  0(  II  x.-z  l|) 
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. 

I 

I 

I 

1 


■ 


we  infer  from  (4.3)  and  (4.4)  that 

(4.6)  ||  dj  -  Gpj  II  =  II  dj  +  6j(t)p,  -  Gp  .  II 

<  II  dj  -  GPj  II  *  i^ft) 

=  0  (  I!  x  -  -  z  II  )  . 

Finally  observing  that  by  (4.5)  and  Lemma  1 

(4.7)  djPj  =  djPj  +  5 j ( t )  u  for  j  sufficiently  large 
we  obtain  the  statement  of  the  lemma  from  (4.2)  and  (4.6). 


In  order  to  obtain  estimates  for  the  remaining  terms  on  the  right  hand 
side  of  (4.1)  we  observe  that  by  (3.8) 


(4.8) 

and 

(4.9) 


t  J-  - 1) 

yi 


,r- 1 
w .  G  w  • 


w  .q  . 

rj 


1,  (i.  )_<9j*l-yj)'G'1(9M-yj) 


llqi  -  a.p  • 

(Y  -  l)  — I - J-J- 


{gj+i'y.i)'Hj+i(9j+ryj) 

^9j+i  iy;f_Hj+iGHj+i^  9j+i 


wjqj 


uVGu.  j 

-  J  =  (1  -—  ) 

9  ^9j+l  "  yj^ Hj+l^9j+l  "  yj^ 


-  y 


Because  no  a  priori  upper  bound  for  these  terms  is  available  we  introduce  the 
following  restriction  on  the  choice  of  the  parameter  t  . 


Condition  2 


For  every  j  the  parameter  t  is  determined  such  that 


1, 


il  gjtl  ii2 

ii  ij.i  ii2 

j 

^9j+l  yj^  ^j  +  l^9j+l  "  yj ^ 

9j*iHj*i9j+i 

I'  sj+l " 2 

||  sJ+i»2 

(9j+i 

9j+lHj+l9j+l 
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'  i 

r  •  ** 

-  ”  6  11 9  j ' 


6  11  9j 


II  yi  H  ( it  yj  ii  +  ii  qj+1n) 

'  q.i  +  i  yj  ^  Hj  +  i;  gj  +  i  yj  ^ 


6  llgjH 


.  j’1  11  Hj+lyj  "  ^  11 H 1~^ j  11  +  I|SJ+1 1 


Yj 


^gj  +  l  ’  Yj  ^  Hj+l^9j+l  ’  yj  ^ 


u6  gj 


where  is  an  arbitrary  positive  constant. 


Since  it  follows  from  (3.7)  and  (2.30)  that  y,  =  0  if  t  =  1  , 

vJ 

there  is  tj  <  1  such  that  every  tj  <  t  <_  1  satisfies  Condition  2. 


Lemma  5 


If  Condition  2  is  satisfied,  then 


1  wiG  lwi  °i+lgi+lG  lgi+l 

(J__1)J - 1  -  (1-Y.)  -  At1  Jl1 - HI  =  O(llx.-zll) 

j  Wjqj  9j+lPj+l 


and 

2  uiGui 

(Yi  -  1)  II  q  “  ot-jP-j  ir  -A-A  -  (1  y 
J  J  J  J  w'-q- 

J  J 


L)  pj+iGpJ+i 

9  °j+i9j+ipj+i 


=  0  ( llXj  -  z  I 


Proof: 

Observing  that  p^+1  =  Pj+^Sj+^  we  obtain 

(gj*i~yj)'G'1(g]*i~yj)  .  g.i+iG'lgj-n 
(9j+i'yj)Hj+i(9jtryj>  9i+isj+i 

WLSh _ + 

(9jtryj)'Hj+i(9j+ryj)  9j+iVi9j+i  (9j+i'yj)'Hj+i<Vi'yj> 

By  (4.8),  Condition  2  and  part  i)  of  Lemma  1  this  implies  the  first  statement 
of  the  lemma.  Similarly  the  second  statement  follows  from  (4.9),  the  relation 
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yrWiPVi  »w>ji 

qj  +  l  "  y  )  ^  Hj  +  1  n.;*l  ’  Vj  ^ 


Pj+lGPj+l 

j+l9j+lPj+l 


Sj+lG$j+l 


si+iGsj+J  +  '  2sj+iGHj+iyj 

1 9j+i "  yj^Hj+i(9j+i " y.^  qj+isj+i  ^9j+i 'yj^'Hj+i^9j+i -yj 


Condition  2  and  part  i)  of  Lemma  1. 


he  next  lemma  shows  that  the  sequence  {  Xj  }  converges  sufficiently 
fast  to  z  to  make  the  sum  over  all  numbers  II  x- -  z  II  finite. 


Lemma  6 


Let  Condition  1  be  satisfied.  Then  the  sum 


(4.10) 


I  II  x.  -  z 
j=0  J 


is  finite. 


Proof: 


By  Taylor's  theorem  there  is  a  vector 

z  •  €  {  x  1  z  +  A  (x  .  -  z)  ,  0  <  A  <  1  } 

J  J 

such  that 

F(Xj)  =  F(z)  +  j  (*j  "  z)'G(Zj)(Xj  "  z)  . 

From  this  equality  we  deduce  the  relation 

(4.11)  u  II  Xj  -  z  II2  <  2(F(Xj)  -  F(z) )  £  n  II  x.  -  z  II2  . 

Furthermore,  it  follows  from  Lemma  2  that 

,  (9jpi>2 

f(x  ,)  ;  F(x  )  -  . 

Subtracting  F(z)  on  both  sides  of  this  inequality  and  using  (4.11)  and 
Lemma  1  we  obtain 
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(4.12) 


F(x.  ,)  -  F( z)  i  F(x  )  -  F ( z ) 


(gjPj) 


=  (F(x-)  -  F(z)J  1 - ^ - | 

l  J  '\  2(F(xj)  -  F(z))n  ) 

,  \(  (gjPJ)2  '■ 

1  F(x.)-F(z) \\l--U- - ?  1 

V  J  '  \  riZ||Xj-zir 

/  \(  2  (9jpj)2 

1  F(x.)-F(z)  1-^2  -2-2-z  ■ 

[  3  '\  n2  II  9j  ir  ' 


(4.13) 


4,  =  1  - 


UgjPJ 

n  II  gj  II 


we  infer  from  (4.12)  that 


(4.14) 


F ( xj+i )  "  F(z)  <  (F(x0)-F(z))  II  ^ 


For  every  j  let  j(k)  denote  the  number  of  elements  i  in  the  set 


{0,1,.. . ,j)  for  which  the  inequality 

II  gill 

(4.15)  — 


>  2  y. 


holds.  Then  it  follows  from  Lemma  3  that 


J  II  9 


2  ut  J(k)  <  I  - —  <  (j  +  1)  Po 

i=0  q '.  p . 


(4.16) 


J(k)  (J  +  1)  • 


If  we  set 


,2  =  i 


v  \2 
2  Uo  n 


then  (4.13),  (4.15)  and  (4.16)  imply  that  at  least  one  half  of  the  numbers 


5 i  =  0,1,..., j  ,  are  less  than  or  equal  to  C  . 


-  28  - 


I 


There  fore , 


(4.17) 


rr  r.  <  -j+1 


Combining  (4.11),  (4.14)  and  (4.17)  we  see  that 


This  shows  that 


xj-z  II2  <£  (F(Xj)  -  F(z) )  <  rJ  +  1  (F(xq)  -  F(z) )  \ 


Xj  -  z  II  =  0(>J)  for  some  0  <  >.  <  1 


from  which  it  follows  that  the  sum  (4.10)  is  finite. 


In  the  next  lemma  we  establish  the  boundedness  of  the  sequences  {  Sj  •  , 

and  (W>- 

Lemma  7 

Let  Assumptions  1  and  2  and  Conditions  1  and  2  be  satisfied.  Then  the 

following  statements  hold 

i)  The  sequence  {  r.  }  is  bounded. 

J 

ii)  The  sequences  {B.(y.)}  and  {H  -  ( y  - ) }  are  bounded. 

J  J  J  J 

iii)  II  ■Yj_1PJ*gj  “  GPj  II  -*■  0  as  j  -*■  °°  . 


Proof: 

i)  It  follows  from  the  equality  (4.1)  and  Lemmas  4  and  5  that  there  is 
a  constant  u-y  such  that 

(4.18)  r  <  5  ♦  „  II, -ZM  t  . 

"j") 

Since  aj  =  dj q j  /  dj Pj  and  by  (3.6)  djpj  >  0  is  bounded  away  from  zero,  we 


7 


,4.W)  /  PjGPj  -  2  jPjGqj 


(d'.n.)2 

J  J-  +  (piG-di)p,  A 
djpj  \djpj 


-  2(PjG-dpq. 


d_h 

d?j 


=  0  (  II  d.  -  Gp.  II  )  =  0  (  II  X.  -  z  II  )  , 

J  J  J 

where  the  last  equality  follows  from  (4.6). 


Observing  that  by  (4.2),  £,  >  1  and  £.  >  1  /  wiq ■  we  deduce  from 

J  J  J  J 

(4.18)  and  (4.19)  the  inequality 


*J+1  -1  ^  v7  I'  xj  -  z 'I  +  0 


Xj  ~  Z 


Wjqj 


<  (  1  +  U8  llXj  -  z  II  ) 

=  II  (  1  +  Uo  II  X.  -  z  II  )  , 

0  1=0 

where  up  is  a  suitable  positive  constant.  Since  by  Lemma  6  the  sum 


Z.  II  X.  -  z  II 

j=l  3 

is  finite  this  implies  that  <  ug  for  some  constant  pg  and  all  j  >  0  . 


ii)  For  each  j  ,  the  matrices  B.(y.)  and  H.(y.)  are  symmetric  and 

J  J  J  sj 

positive  definite.  Hence  all  their  eigenvalues  are  real  and  positive.  Since  by 

definition,  is  equal  to  the  sum  of  their  eigenvalues  it  follows  from 

J 

part  i)  of  the  lemma  that  the  sequences  and  are  bounded- 


iii)  By  part  i)  of  the  lemma  the  sequence  {l/w'.q.} 

J  J 

fore,  (4.19)  implies  that 


is  bounded.  There- 


l 

I 


30  - 


,4.20) 


V  j'V'c,:qj-  ■jpj)  "jG,|j 


0  ( II  Xj  -  zl 


Since 


j  0  for  all  j  ,  it  follows  from  (4.1),  (4.20)  and  Lemmas  4,  5 


and  6  that  the  sum 


”  j  pjGp.i  *  U-l'"/  9i'G'lgi  .  ,) 


is  finite.  Hence, 


Js0\ 


p;Gpru4plpiG~S 


2  as  j 


Yj-lpj9jPj 


which  by  (4.2)  impl ies 


Ll-lpj9jpj  •  Gpj  11  *  0  as  j 


Using  the  above  lemmas  we  can  now  prove  the  main  result  of  this  section. 


Theorem  2 


Let  Assumptions  1  and  2  and  Conditions  1  and  2  be  satisfied.  Then 


xj+r z 

II  Xj  -  Z  || 


0  as  j 


Proof: 


It  follows  from  (3.1)  that 


(4.21) 


pJGPj*pys(Zj)-6)Pj 


Since 


(4.22) 


ti-iVjpj  *  (P5G-  Yj-iBjgi)pj  *  pj(G<zj>  -  G>p.i 


G(Zj )  -  G  II  <  II  G(Zj )  -  Gj  II  +  II  Gj  -  G  II  -*■  0  as  j  -  » 
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l'f  ,  ■, 


and  by  part  iii)  of  Lemma  7 

(4.23)  II  -  Gpj  II  -  0  as  j  • 

and 

Yj-lpjgjPj  - 

we  infer  from  (4.21)  that 

(4.24)  11  °jsj  11  ‘  “J'J-l 

Furthermore, 

(4.25) 


-h _ g  — i-i- 


gj  ii  ii  gj  ii 


for  j 

sufficiently 

large 

Yj-1  " 

1  as  j  -  *  . 

g.i 

c  Sj 

.  g  -  y  j-1  ' : 

.1 9j  II 

Yj-1  "  9j  1' 

Y j  ■  1 "  gj  11 

gj 

.«  SJ 

i  ,lsi" 

+  3  J 

and 


"V  Yj-i11  sj" 

II  94 


Yj-1 " 9J " 


- 1 pj g j  *  Gpj 11  =  Yj-ipj 11  gj 


sj 


Jj"  Y0-l"  gj  11  11 


Therefore, 


(4.26) 


Sj 


gj"  Yj-l"gj 


9j  "  " 


Yj-lDjgj  *  GPJ 


Observing  that  by  part  iii)  of  Lemma  7  the  sequence 

j  1  )  .  f  11  s.i  11  \ 

1  Yj°j"  "J  "  1  Yo  "5j"f 

is  bounded  we  deduce  from  (4.23)  through  (4.26)  that 

9,- 


-  G 


°jSJ 
II  n 


-*•  0  as  j  -  “  . 


In  conjunction  with  (2.5),  (2.7)  and  part  i)  of  Lemma  1  this  completes  the 
proof  of  the  theorem. 
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In  the  following  lemma  it  is  shown  that  the  sequence  . j  converge 


to  one. 


Lemma  8 


Let  Assumptions  1  and  2  and  Conditions  1  and  2  be  satisfied.  For  every 
choice  of  :•  l  and  62  such  that  61  62  _•  0  and  ^  ^  *  0  the  fol¬ 

lowing  statements  hold. 


i )  0  ■  1  j  =  0,1,2,. . . 


'1  -Yj 


0  (max  { 


II  X.  -  2  II' 


Xj  -  2  l|  )  . 


Proof : 


It  follows  from  (2.41)  and  (2.39)  that 


(4.27) 


,  -iVj  *  ^VjVeijV  /» j<ij) 

BldjPj  +  e2djHjdj 


=  ,  _  s2(dyy  /wjqj 

s1ajpJ  +  e2d‘.H]dj 

Because  6«( ,dip  .  +  -d  • )  •  0  the  above  equality  proves  the  first  part  of 

*  J  J  ^  J  J  J 

the  lemma.  Furthermore,  the  relation 

S0  d'.H.d.  <  1  Sid'.p .  +  Sod’.H  .d  • 

2  J  J  J  -  1  rj  2  j  j  j 

shows  that  it  suffices  to  prove  the  second  part  of  the  lemma  for  the  case 
pj  =  0  and  ^  =  1  •  With  this  choice  of  parameters  we  obtain  from  (2.41) 


the  equality 


(4.28) 


i  ,  jqjpjdjH.jdj  .  (dj.pi)2  +  (d^j) 


(djPj)* 


(W‘ 


r’iqipi  ? 

1  +  ,)2 

^  J  J 


'  / 


Because  l/wU|- 


j9?j  lpjGpj  * ( ■  j-i' j9jpj  ■  Wj 


it  follows  from  (4.28)  and  parts  i)  and  iii)  of  Lemma  7  that  there  is  s  • 


constant 


0  such  that 


(4.29) 


,  i  +  iO-  (d'.q.)2  ,  j  =  0,1,2 . 

Yj-1  J  J 


Finally  observe  that 


(4.30) 


<)jqj  ’  i  djqj  *  '  Sj<t)Pjqj 
<  '<gj '  9j»d'qj'  t  . 


9j*iqj 
II  a^s.  II 

J  J 


Mt)  1  . 


Since  by  (3.1)  and  part  ii)  of  Lemma  7 


PJG(ZJ)Pj  < 


,hjV 


» gj 11 


and  by  (2.30)  and  (2.29) 

1  6j(t)  I  =  0  (  II  xj+1-Xj  II  )  =  0(  ||  Xj-z  II  )  , 

it  follows  from  (4.30)  and  part  i)  of  Lemma  1  that 


(4.31) 


djC,j  <  -  0( 


11  Xj  +  l‘Zl1  + 
11  Xj  '  Z  11 


Xj  -  z  I.  )  . 


By  Theorem  2,  the  right  hand  side  of  (4.31)  converges  to  zero  which  by  (4.29) 
implies  that  the  sequence  (1/fj}  is  bounded.  Hence  we  obtain  from  (4.29) 
the  equal i ty 

;  i  -  ,j  .  o(,d’qj)2)  . 


•/ 


34  - 


> 


(H  .1  / .  i  ,  ■  0  as  i  •  this  impl  ies 

i-1  =  • 

in  ,  •  t  4.  1  this  relation  completes  the  proof  of  the  lemma. 

i  ml  theorem  it  is  shown  that,  for  j  sufficiently  large, 
between  zero  and  one  satisfies  Conditions  1  and  2. 


neorem  i 

/apt  ions  1  and  2  and  Conditions  1  and  2  be  satisfied  and  suppose 
•  *  •  *  0  and  •  j  +  ^  *  0  •  Then 

*ne  sequences  and  ■  H j }  are  bounded. 

ii )  'j  *  1  as  J  -  ’  • 

"nere  is  j  such  that,  for  j  jQ  ,  every  t  ,  0  <  t  <  1  , 

satisfies  Conditions  1  and  2. 


Proof: 


Because  it  follows  from  Lemma  8  that 


Yj  -  1  as  j 


(4.32) 

the  first  part  of  the  theorem  follows  from  part  ii)  of  Lemma  7.  In  order  to 
prove  the  second  part  of  the  theorem  we  deduce  from  (4.21)  the  relation 

SjPj _ 


1  'ISjUPj  («(Zj>  -  G)Pj  +|lsjlt(p^.G- Vj-iOjgjI'Pj 

which  by  '4.32),  (4.22),  part  iii)  of  Lemma  7  and  the  boundedness  of  the  se¬ 
quence  II  s  •  II  /gtp.  j  implies  that  v,  *  1  as  j  -  •  . 

J  J  J  J 


Because  t.  =  d'.q./d'.p.  it  follows  from  (3.11)  and  (4.7)  that 
J  J  J  J  J 

h  ■  °<v  • 
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Thus  Lemma  8  and  part  i  of  Theorem  3  imply  that  every  t,  0  t  1  , 
satisfies  Condition  1  for  j  sufficiently  large.  Furthermore,  using  (3.7,, 
(4.7)  and  the  fact  that  9j+1Pj  =  0  ,  it  is  not  difficult  to  verify  that 
there  is  c  >  0  such  that 


(4.33)  min  <  ||  — Eitl_  -  — jj  ,  j!  - jj  )  >  e  ,  1  =  0,1,2,... 

I  11  II  n  .  .  II  II  n  .  .  II  11  11  II  SJ  -  II  II  V  .  II 


II  yj  il  lly  j  II 


Because  by  (4.5) 


I  6j(t)  !  =  0  ( II  Xj  -  z  II  )  ,  0  t  1  . 

it  follows  from  (3.7)  that 

il  yj  II  =  o(ii  x.  -z  ii2  )  , 

we  deduce  from  (4.33),  Lemma  1,  and  the  first  part  of  the  theorem  the  relation 


(4.34) 


"  9j  +  i»  _  »  gj-el11 

^gj  +  l"yj^  Hj  +  l^gj  +  l  "  yj^  9j  +  lHj  +  lgj  +  l 


=  0  min  (  1 , 


l|xj-zl1' 

"Xj+l-zl 


Since  by  Lemma  8 


1  -  yj  |  =0  max  |  II  Xj  -  z  II2  , 


11  xj+l  -  z  11  1 
II  x.  -  z  II2  jf 


we  obtain  from  (4.34)  the  equality 


J  <aj*i'yj,Hj+i(9j+i‘yj)  9j+iHj+i9j+i 


I  =  Olmax  {||Xj  -  zll  ,  (I x ' 


Using  Lemma  1  and  Theorem  2  we  see  that  the  first  inequality  of  Condition  2  is 
satisfied  for  all  t,  0  <  t  _<  1  ,  if  j  is  sufficiently  large.  Since  a  com¬ 
pletely  analoguous  argument  shows  that  the  remaining  inequalities  are  also  sa¬ 
tisfied  for  all  t,  0  _<  t  <  1  ,  and  all  sufficiently  large  j  this  completes 
the  proof  of  the  theorem. 
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